Above-water monitoring of swimming pools

ABSTRACT

An above-water system provides automatic alerting for possible drowning victims in swimming pools or the like. One or more electro-optical sensors are placed above the pool surface. Sequences of images are digitized and analyzed electronically to determine whether there are humans within the image, and whether such humans are moving in a manner that would suggest drowning. Effects due to glint, refraction, and variations in light, are offset automatically by the system. If a potential drowning incident is detected, the system produces an alarm sound, and/or a warning display, so that an operator can determine whether action must be taken.

CROSS-REFERENCE TO PRIOR APPLICATIONS

This is a continuation of U.S. patent application Ser. No. 12/479,744,filed Jun. 5, 2009.

Priority is claimed from U.S. provisional patent application Ser. No.61/084,078, filed Jul. 28, 2008, entitled “Above-water System forAlerting of Possible Drowning Victims in Pools of Water”, the entiredisclosure of which is incorporated by reference herein.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of automatedmonitoring of swimming pools, and the like, to detect possible drowningvictims. More specifically, the invention relates to systems which useonly sensors that are above the water line, to alert responsible personsmonitoring a pool of water, by detecting behaviors consistent with thoseof someone who is unconscious or otherwise incapacitated.

Devices for automated monitoring of swimming pools have been known inthe prior art. Such devices have employed video or other sensortechnologies, such as sonar. Examples of such devices are given in U.S.Pat. Nos. 6,133,838, 7,330,123, and 5,043,705, the disclosures of whichare incorporated by reference herein.

The above-described prior-art devices are limited in theirfunctionality, in that all require the mounting of sensors below thesurface of the water. Mounting the sensors below the surface requires amore costly and disruptive installation procedure, requiring the routingof power and data wires underwater, or through the pool walls, back tothe sensor processing hardware. Also, the systems of the prior artrequire extensive or cumbersome calibration methods or algorithms toreduce false alarm rates.

In U.S. Pat. No. 6,133,838, there is described a system using underwatercameras mounted to the walls of a swimming pool. Underwater cameras havean advantage in seeing underwater objects and humans without theobscurations caused by the surface refraction effects at the air-waterinterface. However, the use of such a system involves the cost andcomplications of draining the pool, drilling large holes into the poolwall, installing watertight video camera housings, and excavating behindthe wall to route wires to the cameras.

Moreover, in the above-described system, because the underwater camerasmust be flush with the wall contours, the system has blind spotsimmediately adjacent to the pool walls, especially near the cameras. Theprior art system must accept these disadvantages as the price foravoiding the additional signal processing needed to extract usefulimages if the cameras were mounted above the water surface.

U.S. Pat. No. 7,330,123 discloses sonar devices mounted underwater onthe pool walls, and/or the pool bottom, to scan for objects and humansdisplaying characteristics of interest. These are active sensors, ascontrasted with the passive sensors of the present invention.Pool-mounted active sensors are likely to be accidentally dislodged orblocked by swimmers, thus disabling one or more of the sensors. Thesystem also requires that a person with an active sensor be in the pool,to support calibration of the overall system for different numbers ofswimmers and/or levels of activity.

U.S. Pat. No. 5,043,705 uses a similar active sonar system to scan thesurfaces within the volume of a pool, to generate images from which thesystem can discern objects and humans who are stationary. As in theabove-described patent, its sensors are vulnerable to accidentaldislodgment and/or blockage by swimmers.

The sonar systems of the prior art could not be mounted above the watersurface. The problems of the video-based prior art could theoreticallybe avoided by providing sensors above the pool. However, the prior arthas taught against doing so, because of the intractable problemsencountered.

Specifically, the air-water boundary presents a number of challenges tosensing algorithms and makes it impractical simply to move an underwatersystem to a position above the water line. A water surface has smallsurface waves, creating a roughened water surface, akin to a rough oceanon a small scale. This surface acts as a series of small areas withslightly different refraction properties, producing the fractured anddistorted view seen when observing objects underwater. Objects appeardisjointed to an observer and often are missing segments due to changesin surface refraction distorting and breaking up the sensed image ofunderwater objects.

Moreover, varying water quality and lighting conditions alter the sensedimage of the water being monitored, adding to the difficulty of usingabove-pool sensors.

Sensors mounted underwater do not have to deal with glare on the surfaceof water, or surface refraction. Further, underwater sensors areoriented to resolve the up and down motion of swimmers, while abovewater sensors are usually positioned at a more oblique angle, and mustuse passive ranging techniques to monitor motion in the criticalvertical axis. For these reasons, it is impractical simply to move anunderwater system of the prior art to a position above the water line.

It is the purpose of the present invention to overcome the aboveproblems, and to provide a practical system and method for monitoring aswimming pool from above the pool. The present invention provides a newand useful above-water pool-monitoring system which is simpler inconstruction, more universally usable, and more versatile in operationthan the devices of the prior art.

SUMMARY OF THE INVENTION

The present invention provides an automated pool monitoring system whichincludes sensing objects through the air, the air-water interface, andthe water itself. The present invention uses passive electro-opticalsensors that are mounted only above the water surface, and near the poolperimeter.

The present invention uses passive ranging techniques to estimate thethree-dimensional location of objects on or under the surface of thepool. Further, the invention uses spectral processing to account forvariations in lighting and water quality conditions, and uses spatialprocessing to untangle the distortions introduced by the roughened watersurfaces. Finally, the present invention employs one or more polarizinglenses and/or special spectral filters to overcome glare, shadows andthe like.

Together, the above-described procedures overcome the limitations whichhave prevented devices of the prior art from being moved from below thewater line to a position above the pool. The present invention overcomesthe effects of surface distortions to reconstruct an undistorted view ofunderwater swimmers.

The present invention alerts responsible persons monitoring a swimmingpool concerning the possibility that someone may be drowning. Theinvention provides an alert in the form of a sound and a visual display,enabling the operator to assess the location which caused the alert. Theoperator can then determine whether action must be taken, and turn offthe alert from any remote display.

The system includes one or more electro-optical (EO) sensors mountedabove the surface of the pool. The EO sensors are mounted at a heightabove the water surface that provides an adequate angle of view thatincludes a significant portion of the water surface and the pool bottomsurface at a resolution consistent with the overall system fidelity.

The process of the present invention comprises at least three basic,interrelated parts, namely 1) spectral processing, 2) spatialprocessing, and 3) temporal processing. The spectral processordecomposes each digital image into principal components, for the purposeof enhancing contrast, or signal-to-noise ratio. The output from thespectral processor is fed to the spatial processor, which searches forparticular, tell-tale shapes in each image. The output of the spectralprocessor is fed into a temporal processor, which analyzes a sequence ofimages, especially a sequence of images containing the shapes ofinterest, to detect movements (or lack thereof) that may indicatedrowning.

In addition to the above, the system is programmed to compare sequentialimages to determine which pixels, if any, are artifacts due to glint.Such pixels can be discarded to improve the quality of the images.

The present invention therefore has the primary object of providing asystem and method for monitoring a pool, and for warning of thepossibility that someone is drowning.

The invention has the further object of providing a system and method asdescribed above, wherein the system uses passive sensors which aremounted above the surface of the pool.

The invention has the further object of providing a system and method asdescribed above, wherein the system overcomes the problems ofdistortions inherent in viewing objects in a pool, from a viewpointabove the surface of the pool.

The invention has the further object of reducing the cost, and improvingthe reliability, of systems and methods for monitoring pools forpossible drowning victims.

The reader skilled in the art will recognize other objects andadvantages of the present invention, from a reading of the followingbrief description of the drawings, the detailed description of theinvention, and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 provides a perspective view of an above-water system for warningof possible drowning victims in pools of water, according to the presentinvention.

FIG. 2 provides a schematic and block diagram, showing the hardwareconfiguration for the system of the present invention.

FIG. 3 provides a block diagram illustrating the architecture of thesystem of the present invention.

FIG. 4 provides a block diagram illustrating the processing algorithmsused in the present invention, for detecting possible drowning victimsin a swimming pool.

FIG. 5 provides a flow chart illustrating the steps for performingspectral processing for the system of the present invention.

FIG. 6 provides a flow chart illustrating the performance of stereoprocessing for the system of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description of an above-water system for monitoringpools for possible drowning victims, reference is made to theaccompanying drawings forming a part thereof. These drawings show by wayof illustration, a specific embodiment in which the invention may beimplemented. Other embodiments may be utilized and structural changesmay be made without departing from the scope of the present invention.

In this specification, the term “video” is defined as a series of timesequenced electro-optical CEO) images within a portion of the bandwidthof wavelengths from infra-red to ultraviolet energy. The EO sensors maybe mounted on rigid poles, walls, or ceilings, or any combinationthereof. The sensors receive video images of the pool surface includingimages of humans and objects within the water volume, at or below thesurface.

The EO sensor housing may include a pair of apertures at a knownseparation distance providing stereoscopic images of the field of view.The stereoscopic images improve the accuracy of the estimated range ofthe targets being viewed, allowing for better determination of the depthof the humans being tracked in the field of view.

The EO sensors may include polarizing lenses and/or special spectralfilters that transmit only certain portions of the electromagneticspectrum. The polarizing lenses and/or filters aid in reducingreflections which obscure details of features within the image of thewater within the field of view of the sensor.

The present invention overcomes the effects of 1) bright reflections, orglare, caused by the sun or artificial lights, 2) refraction of lightcaused by large or small ripples in the water, and 3) light refracted bysmall bubbles caused by agitation of the water.

A light intensity meter that measures the amount of light in the fieldof view may be co-located with each sensor housing. The light intensityinformation can aid the signal processing algorithms in determining therange of color contrast that is available, which, in turn, improves theaccuracy with which one can detect which contours and/or colors areedges of the human form. Moreover, the system will alert wheninsufficient light is available, based on the light intensity meterreadings, and will inform responsible persons that the system should notbe used at that time. The system can then notify responsible persons,when the light level is again sufficient for video processing.

The video images captured by the system of the present invention aredigitized and processed using computer algorithms to identify whichobjects within the field of view are humans, and to determine thethree-dimensional coordinates of one or more points characterizing thelocation of each human. The digitized images are processed to removeadditional remaining obscurations of feature details within the image.Sequential processed images are compared to determine if any humanwithin the water volume is displaying the characteristics of a possibledrowning victim.

For example, drowning characteristics to be detected could include aperson exhibiting a downward vertical velocity with minimal velocity inthe two orthogonal directions and minimal movement of arms or legs. Ifsuch characteristics are observed, the system will execute an alertingalgorithm whereby a signal is sent to all active monitoring devices.That alert includes a display that indicates the location of thepossible victim relative to various pool features (such as the poolperimeter, lane-marker tile patterns, etc.).

Portable alerting devices, to be worn on the wrist and/or around theneck of an operator, may be included as part of the present invention.Any active person monitoring the pool has the ability to observe thealert location in the pool to determine if the situation requiresaction. If the pool is being monitored remotely, the operator can viewthe live video images of pool from any of the EO sensors and make thesame judgment regarding whether it is necessary to take action, orwhether the alert should be turned off.

An embodiment of the system may include a connection to the Internet toallow for two-way communication between the user and the systemprovider. Each user system will download to a central processing siteinformation such as: imagery of the pool scene to help withinitialization and calibration of the system installation, and thetime/location of alert events. The central processing unit will uploadto the users information such as: calibration factors duringinitialization, any software upgrades/updates, and/or traininginformation.

FIG. 1 provides a perspective view of the system of the presentinvention. In the system illustrated, there are two passiveelectro-optical (EO) sensors S1 and S2, mounted above the water level ofthe pool P. The sensors are therefore positioned to observe the entirevolume of water in the pool. The number of sensors is not limited totwo: in practice, additional sensors could be present.

FIG. 2 provides a schematic and block diagram of the hardware used inthe present invention. Video images are received by the EO sensors S1and S2. Polarizing lenses 2 and light filters 3 may be placed in frontof the sensors to restrict the light reaching the sensors to a narrowband of the optical spectrum. A light intensity meter 12, for sensingthe amount of light present in the field of view of the sensor, may beco-located with each sensor. Knowing the light intensity aids the signalprocessing algorithms in identifying contrasts that are identifiable asthe edges of human bodies.

The image is converted to a digital signal in converter 5. In practice,the converter may be located within the sensor units S1 and S2. Thedigital signal is then transmitted to central processor unit (CPU) 6 andto dynamic random access memory (DRAM) 7. The CPU can be amicroprocessor, or its equivalent.

The CPU performs processing algorithms to discern: a) humans who are inthe water, b) whether the observed humans are showing behaviorconsistent with possible drowning, and c) how to indicate an alert tothe monitoring person(s) or operator of the system.

Long-term memory device 8 stores processed and raw data, to allow forretrieval at a future time. All digitized image data can be transmittedto the CPU by way of either cables or a wireless network. Power supply 4provides power to the EO sensors, and to the CPU and monitor, and couldrepresent either a distributed source or local sources.

Central computer monitor 11 displays scene imagery, showing the scene ofthe pool as well as system status and any alerts and the zone in whichthe alert arose. Alert information may also be sent via a wirelessconnection 9, to a distributed network of devices 10, that sound analarm, vibrate, and display a zone identification where a possibledrowning event may be occurring.

Each of the distributed devices 10 has the ability to send back to theCPU an override signal if the person monitoring the pool determines thatno action is needed. An Internet connection 13 can also be provided asanother means for transferring data, relating to identified events andsoftware upgrades, between each pool monitoring system and the systemprovider.

FIG. 3 shows the functions performed by the system of the presentinvention, in detecting possible drowning victims. Each of theillustrated functions is performed by one or more of the hardwarecomponents shown in FIG. 2, and/or by the CPU. The functions representedin FIG. 3 are together called the drowning detection segment, asrepresented in block A2.

Block A2.1 represents the Sensor Subsystem components. The primarysensor component, Block A2.1.1 represents the functions performed by anappropriately selected, commercially available video camera capable oftaking and digitizing images at a rate of more than 2 images per second,at a resolution such that one pixel covers a small enough area toresolve human features such as a child's hand.

The image received by the primary sensor component may be filtered usinglenses, to receive only energy of a single polarization, and/or one ormore, specific, monochromatic bandwidth(s) of energy.

A sensor site may include more than one sensor at the same location, thesecond sensor being termed a secondary sensor component, as representedin Block A2.1.2. The secondary sensor component can be of the same typeas the primary sensor component, and may have essentially the same fieldof view. The secondary sensor component can be configured to receivedifferent types of polarized/filtered energy. The secondary sensorcomponent could also view the scene from a different location, allowingfor stereoscopic image processing.

All data received by the sensors must be calibrated with respect to thespecific conditions under which the electro-optical image is received.Block A2.1.3 represents a calibration component. Calibration can beperformed by comparing the amplitude, specific reflectance bandwidth,and resolution of known, constant features, that are printed, etched orotherwise made part of the protective lens for the sensor. Data from thelight intensity meter also may be used in this module to aid inachieving the best contrast of the humans beings monitored. Imagesreceived of the pool scene can then be adjusted under the instantaneouslighting conditions to be consistent with the expected parameters ofsubsequent image processing algorithms.

If the calibration parameters indicate that the system is not receivingvideo images within the expected ranges, due to conditions such asinsufficient ambient light, processing performed within the illuminationcomponent A2.1.4 of FIG. 3 will indicate the out-of-tolerance condition,and will alert the user that the system is not functioning.

The illumination component uses the output of a light meter, or“incident light sensor” (ILS), or its equivalent, to make a decision,based on the amount of light received, whether to continue theprocessing. When the amount of light received is acceptable, but above agiven threshold, the system can be programmed to weight the components(i.e. the component colors) of the image so as to yield optimum results.

The environmental sensor component, represented in Block A2.1.5 of FIG.3, monitors variations in the scene that may change due to seasonal orintermittent weather conditions. One example is the periodic imaging ofa constant, known object within the pool scene itself to augment thecalibration of the image data received by the sensors. The incidentlight sensor, discussed above, may be used in conjunction with thiscomponent.

Block A2.2 of FIG. 3 represents the processing subsystem of the presentinvention. The data acquisition component, Block A2.2.1, includes meansfor receiving the digitized video images at a known rate. Each digitizedimage frame is a matrix of pixels with associated characteristics ofwavelength and brightness that are registered to the physical locationwithin the scene as it is projected from the pool area. Each image frameis tagged with a time stamp, source, and other characteristics relatingto the acquisition of that frame.

Within Block A2.2.2, the digitized image frames are then filtered toremove additional obscurations through signal processing methods suchas, but not limited to, averaging, adding, subtracting image data of oneframe from another, or by adjusting different amplitudes relating to theimage contrast, brightness, or spectral balance. The detection thresholdcomponent A2.2.3 analyzes the processed image frames to detect whichpixels within the registered frame are humans, and to determine thephysical location coordinates of a representative point or points on thehuman.

The detection analysis component, represented in Block A2.2.4, comparesthe images within a specific time sequence to determine if the humansidentified within the scenes are exhibiting behaviors consistent withthose of a person who is apparently not moving or who has begun to sinktoward the bottom of the pool. Such persons could be unconscious andcould possibly be in danger of drowning.

Within Block A2.2.4, several other tests on the perceived behavior ofany detected human are executed to reduce the number of false alarms.For example, a person standing on the pool bottom with his or her headabove water would match the criterion of a non-moving swimmer. However,by also discerning from the images that the person's head is above waterwould indicate that no alert should be generated.

Block A2.2.5 represents the logging component, which simply stores thetagged image frames in random access memory (item 8 of FIG. 2) in theas-received and post-processed formats along with records of specificdiscrete, unique, noteworthy events, such as alerted events, ornear-alert events, for possible subsequent diagnostic reviews.

If an alert condition is detected, the system then executes a procedurefor activating audio, visual and vibrating stimuli to notify themonitoring person(s). Because the system knows the 3-dimensionalcoordinates of the targets, a zone within the pool area established as agrid overlay translates into unique identifiers for each zonecorresponding to a specific location within the pool.

The alert signal will be sent to all alarm devices for that poolindicating the zone where the event is taking place. At a minimum, thealert device will include a large computer monitor (item 11 of FIG. 2)with a plan view image or rendering of the pool area and a flashingsymbol in the corresponding zone where the event is occurring. Portable,distributed alert monitoring devices (such as item 10 of FIG. 2) couldalso be worn on the wrist or around the neck of a monitoring person.These devices would receive wireless signals from the system (asindicated by item 9 of FIG. 2) which would display similar informationas displayed on a central monitor.

If the person monitoring the pool determines that the alert does notrequire action, i.e. if it was a false alarm, the person can cancel oroverride the alert through either by direct input to the central system,or by wirelessly transmitting an appropriate signal through a portablewireless device. If the alert is not overridden within a specified timeperiod, the alert would also notify management personnel within thevenue (through item 11 of FIG. 2). If an alert system is determined tobe an actual drowning event that could require further emergencytreatment, the system could notify local emergency responders through asystem of manual or automatic processes.

The Infrastructure Subsystem components, represented in Block A2.4 ofFIG. 3, include the power component, represented by Block A2.4.3, forsupplying power to the sensors (items S1, S2 of FIG. 2), to the CPU andmemory devices (items 5-8 of FIG. 2), to the central computer monitor(item 11 of FIG. 2), and to any wireless transmitting devices (item 9 ofFIG. 2) connected directly to the central unit. Any portable alarm alertdevices are preferably powered by internal batteries.

The Communications Component, represented in Block A2.4.2, includes thealgorithms by which the alert information is formatted to communicatewith the specific alerting devices for a specific system installation,including computer monitor (item 11 of FIG. 2) and any wirelesscommunication devices such as item 9 of FIG. 2.

FIG. 4 provides a flow chart showing the data processing functionsperformed so as to detect swimmers above, at, and below the water'ssurface. The air-water boundary requires the removal of surface effectsto isolate properly objects which are underwater, and to determine thelocation of the water's surface and thus determine whether an object isabove or below said surface.

The images are acquired in Block 4000, and the constituent colors areextracted, in Block 4001, in order to correct each image from colorcalibration tables represented by Block 4002.

The views from each camera are slightly different, and thus twodifferent cameras, which are purportedly identical, will respondslightly differently to the same input. Therefore, it is necessary tocalibrate the cameras, in advance. Calibration is performed by usingtest colors and images having known properties. One illuminates a scenewith a given level of illumination, and one takes images of the sceneusing the camera to be calibrated. From these data, one can derive atable showing the expected value of each pixel of the image, for eachparticular color and at each particular level of illumination. Suchtables are what is represented in block 4002.

Next, the color corrected images are sent through a series of processingsteps to isolate various spectral characteristics (Blocks 5001-5003)detailed in FIG. 5. FIG. 5 provides an expanded description of what isperformed in block 4003 of FIG. 4.

Next, the specific region of interest is extracted, in Block 4005, andstereo processing functions are performed, in Block 4006, as detailedlater, in FIG. 6, where the first passive ranging estimates arecomputed. The step of ranging includes calculating the distance from thecamera to the object of interest, using multiple cameras and multipleimages, as indicated in Block 4006 of FIG. 4, and which is furthercovered in FIG. 6. Potential targets are extracted from the regions ofinterest in Block 4007 and adaptive thresholds are applied to eliminatefalse targets, in Block 4008. Finally, positive detections are mergedinto a single swimmer centroid, in Block 4009, and final range estimatesare computed in Block 4010.

FIG. 5 provides a flow chart showing the steps performed by the poolmonitoring algorithm during the spectral processing phase (representedby block 4003 of FIG. 4). A series of estimates are made of the colorcovariance, in Block 5000, and are used to determine the principalcomponents of the image, in Block 5001. Next, eigen images areconstructed, in Block 5002, to isolate the colors indicative ofpotential swimmers, and a test statistic is computed, in Block 5003. Thetest statistic helps to determine the thresholds used to differentiateswimmers from the background in the combined ratio color image, in Block5004.

FIG. 6 provides a flow chart showing the steps performed by theprocessor (item 6 of FIG. 2) to determine the range to detected targetsin the water. FIG. 6 provides an expanded description of what isperformed in block 4006 of FIG. 4. Each image is rectified, in Block6000, and sub-pixel registration points are computed, in Block 6001, toenable proper image matching. Next a Snell compensation filter isapplied, in Block 6002, to account for and overcome the surfacerefractive effects of the air-water interface. A spatial estimator iscomputed in Block 6003, and a statistical quality test is performed, inBlock 6004, to determine the effectiveness of the spatial estimator.This process continues until the system has a quality estimate of thespatial extents of the targets in the water, in Block 6005.

In summary, the system and method of the present invention overcomes thetechnical challenges associated with detecting, tracking, anddiscriminating among objects on or under water, using a videosurveillance system which is disposed above the surface of the water.The major problems associated with an above-water system are thefollowing:

a) variations in ambient light levels cause changes in the amplitude ofsignals received;

b) refraction in calm water causes distortion of the images received;

c) refraction and glint, for small and large water waves on the surface,cause distortion of the images received;

d) the images received may be of poor quality, due to a lowsignal-to-noise ratio; and

e) attenuation through the water will be different for differentfrequencies of light, thus causing distortion of certain colorcomponents of signals received.

These problems are addressed by the present invention as follows.

The problem of dealing with variations in ambient light levels is thesubject of illumination component A2.1.4 of FIG. 3.

The variation, over time, of the ambient light level is monitored usingan incident light sensor (ILS), which provides a calibrated measure ofthe radiant energy over specific wavebands of interest. Since thedetection processing methodology of the present invention uses thespectral information in the captured video, it is important to adjustengineering parameters in the multi-spectral image processing chain, asneeded, to compensate for these variations. As an example the localdetection thresholds, for both the spectral image processing and thespatial image processing, would be a function of, and adaptive to, theoverall light level.

Cameras can adjust automatically the gain of an image detector tomaximize image fidelity. Doing so, however, obscures the actual level ofincident light from any downstream processing because the auto-gainvalue is not known for each frame. The present invention instead uses anincident light sensor (ILS), separate from the camera imagers to get alight level reading on a known scale.

The issue of compensation for refraction is the subject of block 6002 ofFIG. 6, which is part of block 4006 of FIG. 4, which in turn is part ofblock A2.2.2 of FIG. 3.

With regard to compensation for refraction in calm water, the presentinvention works as follows. As light passes from one material medium toanother, in which it has different speeds, e.g. air and water, the lightwill be refracted, or bent, by some angle. The common apparent “brokenleg” observed as one enters a pool is evidence of this. Since the speedof light in water is less than the speed of light in air, the angle ofrefraction will be smaller than the angle of incidence as given bySnell's law.

Snell's law can be stated as:

N ₁ sin A=N ₂ sin B

where N₁ and N₂ are the refractive indices of the two media involved (inthis case, water and air), and A and B are the angles of incidence andrefraction. The observed position of an object can be used to derive anangle of refraction, and, since the refractive indices of water and airare known, Snell's law can therefore be used to calculate the angle ofincidence, and hence the correct position of the observed object.

Thus, objects which are under water, and which are viewed from abovewater, will appear to be closer by an amount given by Snell's law, sincethe water acts as a lens, refracts the light and in this case magnifiesthe object with positive power, which for water is about 1.33. For lightreflected from an object and going from water to air, the actual depth Dis 1.33 times the apparent depth.

The system of the present invention therefore applies Snell's law, inreverse, as described above, for each pixel, to correct properly itsposition in three-dimensional space. That is, the system of the presentinvention uses Snell's law to determine exactly how an image wasrefracted, so as to determine the actual position of each pixelrepresenting the object.

The issue of compensation for glint, and for refraction in small orlarge water waves, is illustrated by the same drawings as for the caseof refraction in calm water.

With respect to compensation for refraction and glint for small andlarge water waves on the surface, Snell's law is again used for therefraction component, and frame-to-frame averaging is also used.

Specifically, a sequence of images is collected, and any glint isreduced by polarized optical filters. The de-glinted images are thenstatistically analyzed to determine the pixels in each image that haveminimal distortion due to refraction and are not still obscured by glintthat was reduced through the physical filters. The algorithm discardsthose pixels in regions of an individual image which indicate highdistortion or obscuration creating an area of “no data” for that image.This prevents regions with no useful data from weakening the correlationof the other parts of that image. It also keeps the data from thosedistorted/obscured zones from weakening the correlation with thecorresponding regions in images just prior or later in the timesequence.

A single derived image is reconstructed from the initial sequence ofdistorted images. In this way, one can reconstruct an image using pixelsfrom several images, using only those pixels not affected by the smalland large surface waves. The result has only to account for the normalrefraction, using Snell's law.

In this way, one can reconstruct an image using pieces from severalimages, using only those portions not affected by the small and largesurface waves. The result has only to account for the normal Snell's laweffects compensated for previously.

The system of the invention addresses the problem of improving imagequality as follows. This methodology is represented in blocks 4003 and4004 of FIG. 4, and block A2.2.2 of FIG. 3.

The starting point for image enhancement is the decomposition of thevideo image into its principal components (PC). A given raw image ofvideo is composed of red, blue and green color components. The sum ofthose three components comprises the actual color image seen by aviewer. The three colors for a particular image may in fact containredundant information. Decomposing an RGB image into its principalcomponents is a known statistical method used to produce threepseudo-color images containing all the information in the RGB image. Theinformation is separated so each image is uncorrelated from the othersbut contains pertinent information from the original image. The PCimages are then filtered, using a priori spectral information (i.e. howan expected target should appear in the pseudo color images) aboutfeatures of interest. The extraction method uses a threshold value wherea PC pixel is deemed to be a feature of interest or target if it exceedsthe threshold.

The reason why the three color components (red, blue, green) containredundant information is that the color components, in general, fornatural backgrounds or scenery, are correlated. The object of principalcomponent analysis is to find a suitable rotation in thethree-dimensional “color space” (i.e. red, green, blue) which producesthree mutually uncorrelated images. These images may be ordered so thatthe first PC image has the largest variance PC1, the second image hasthe next largest variance (designated PC2), and the last image has thesmallest variance, designated PC3. The variance, power, is a measure ofthe dispersion, or variation of the intensity values, about their meanvalue. Since PC1, PC2, and PC3 are all uncorrelated with each other, PC1which has the largest variance or power, will generally have the largestcontrast enhancement, while the other two will have less contrast.Furthermore, the orthogonality of the components can be used to aid indiscrimination of particular features.

In particular, looking at functions of the individual intensity valuesof the PC components can allow discrimination and segmentation of theresulting thresholded image.

For example, consider pixel-wise ratios, where R(i,j) refers to the(i,j)-th location in the image array, and define the following:

R1(i,j)=PC1(i,j)/PC2(i,j),

R2(i,j)=PC1(i,j)/PC3(i,j), and

R3(i,j)=R2(i,j)/R1(i,j).

Using properly established thresholds, say T1, T2, and T3, which aredefined by what spectral features are desired to be enhanced, based on apriori knowledge, optics, and the physics of the reflected light, thefollowing spectral filter or statistic, can be used to extract featuresof interest:

Test Image(i,j)=1 for (R1(i,j)>T1 and R2(i,j)>T2 and R3(i,j)>T3)

Test Image(i,j)=0 otherwise, and

Output Image(i,j)=Test Image(i,j)*RGB Image(i,j),

the latter calculation indicating pixel-wise multiplication.

This principal components analysis is performed in blocks 5001-5003 ofFIG. 5, which is part of block A2.2.2 of FIG. 3.

To further enhance the signal-to-noise ratio, a spatial filter is usedon the PC images to enhance spatial shape information. Again, a priorishape filters are used for this. The output of the spatial filter isused to initiate a track of a candidate target and the track is updatedsequentially, in time. The spatial match filter is an optimumstatistical test which maximizes the signal to noise ratio at locationswhere a target or feature is present.

More particularly, the spatial filter used in the present inventionmeasures the correlation between a known shape and the image beinganalyzed. Thus, one must know in advance the shape of the target beingsearched, up to a scale factor. The procedure comprises a patternmatching process, where a known spatial pattern is convolved with aninput image to yield an output of SNR (signal-to-noise ratio) values.

For example, suppose that it is desired to detect a square shape in animage that contains that shape plus added noise. One begins with atemplate comprising a white square in a black image. That is, the pixelsin the square have a value of one (maximum brightness) and the pixelselsewhere are zero (black). Shifted versions of this template are usedto locate the square pattern in the raw image.

To start the correlation processing, the match filter output at thatlocation will be the sum of the pixel-wise product of the template imagewith the raw image. For each template, the sum will be the sum of thepixel values in the image being analyzed, but only in the squarecorresponding to that of the template. Then, a new template is createdin which the square is shifted one pixel to the right, and the processis repeated. The process continues for each row in the raw image.

For targets which may have a particular orientation, all possibleorientations of the template must be considered. So for a rectangulartarget, if the orientation is not known, one must rotate the templateand perform the processing for each orientation. The number of rotationsdepends on the amount of accuracy required. If ten-degree accuracy issufficient, one needs 18 such steps, i.e. each template being rotated byten degrees. The latter would cover all possible orientations in theplane.

The spatial analysis described above yields correlation values for eachcomparison performed. These correlation values can then be used todetermine whether the image being analyzed contains the desired targetshape.

The above-described analysis is covered by block 4004 of FIG. 4, andblock A2.2.2 of FIG. 3.

The present invention addresses the issue of color attenuation throughwater as follows. This issue is covered in block 6002 of FIG. 6, block4006 of FIG. 4, and block A2.2.2 of FIG. 3.

Because wavelengths of light are attenuated to varying degrees throughwater, some are not useful for processing to detect targets underwater.Instead, as mentioned in the prior PC discussion, some add no additionalinformation to the image and can be ignored. Ignoring some of thesewavelengths reduces the processing required to detect and track targetsand speeds up the processing algorithm. It has been found that there maybe little difference between the information content of the blue andgreen wavebands in the imagery, and thus one can variously ignore one ofthem, average them, or sum them to enhance the signal-to-noise ratio ofthe image, without altering the algorithm's perception of potentialtargets.

The process of the present invention can be summarized as follows. Theprocess includes three basic parts, designated as 1) spectralprocessing, 2) spatial processing, and 3) temporal processing. Theseparts are interrelated, insofar as the output of one part is used as theinput to the next.

The spectral processor decomposes each digital image into its principalcomponents, using known techniques, as explained above. The value ofprincipal components analysis is that the images resulting from theprocedure have enhanced contrast, or signal-to-noise ratio, and arepreferably used instead of the original images.

The output from the spectral processor is fed to the spatial processor.The spatial processor searches for particular shapes in each image, bycomparing a particular shape of interest, with each portion of theimage, in order to determine whether there is a high correlation. Theshapes of interest are stored in memory, and are chosen to be relevantto the problem of finding possible drowning victims. Thus, the shapescould comprise human forms and the like.

The output of the spectral processor is fed into a temporal processor,which analyzes a sequence of images, to detect movements that mayindicate drowning. That is, for those images containing shapes ofinterest, such as human forms, the system must determine whether thoseforms are moving in ways which would indicate drowning. The movements ofinterest could include pure vertical motion, or vertical motion combinedwith rotation.

For a given sequence of images, the system can generate a discriminationstatistic, i.e. a number representing the extent to which the sequenceof images contains any of the pre-stored movements indicative ofdrowning. If a sequence of images produces a statistic which exceeds apredetermined threshold, i.e. if the statistic indicates that therelevant movements are likely to be present, an alarm can be generated.The statistic can be generated from a mathematical model representingthe motions of interest.

The temporal processor depends on the output of the spatial processorinsofar as the shapes of interest, detected by the spatial processor,are then analyzed to see whether such shapes are moving in a manner thatwould suggest drowning.

In addition to all of the above, the system is programmed to comparesequential images to determine which pixels, if any, are artifacts dueto glint. Such pixels can be discarded to improve the quality of theimages. This procedure can include an adaptive filter, in that its stepsmay be executed only if obscurations and/or excessive refractiondistortions are detected through pre-set criteria.

For example, suppose an individual is swimming in a pool. The spectralprocessor will enhance the images of the swimmer so that the swimmer canbe automatically recognized as such by the system. Further processing bythe spatial match filter would extract information concerning the size,shape, and location of the swimmer. This information is passed to thetemporal processor, which considers the incoming time series of images,and computes a statistic which indicates the degree to which the motionsof the swimmer match the motions, stored in memory, indicative ofdrowning. If the statistic is above a given threshold, i.e. if thedetected motions of the human form have a high correlation with motionsknown to be associated with drowning, the system generates an alarm.

While the foregoing written description of the invention enables one ofordinary skill to make and use what is considered presently to be thebest mode thereof, those of ordinary skill will understand andappreciate the existence of variations, combinations, and equivalents ofthe specific embodiment, method, and examples herein. The inventionshould therefore not be limited by the above described embodiment,method, and examples, but should include all embodiments and methodswithin the scope and spirit of the following claims.

1. A system for monitoring swimming pools for possible drowning victims,the system comprising: a) a passive sensor, positioned above a surfaceof a pool, the sensor comprising means for receiving an image of thepool, b) means for digitizing images received from the sensor, c) aprogrammable computer connected to receive data from the digitizingmeans, the computer being programmed to analyze the images to determinewhether the images indicate a presence of a drowning victim in the pool,and d) means for alerting an operator of the presence of a drowningvictim, wherein the computer comprises means for compensating forrefraction of images caused by a roughened water surface due to surfacewaves in the pool, wherein the water surface defines a plurality ofareas having different refraction properties.
 2. The system of claim 1,wherein the computer comprises means for analyzing digitized images todetect human forms, and to detect whether said human forms aredisplaying movements consistent with possible drowning.
 3. The system ofclaim 1, wherein there are at least two sensors, and wherein thecomputer comprises means for comparing images received by the sensors soas to compute a distance to an object in said images.
 4. The system ofclaim 1, wherein the computer comprises means for enhancing quality ofimages by extracting principal components of the images, wherein saidprincipal components are non-correlated, and analyzing at least some ofsaid principal components to derive information about the images.
 5. Thesystem of claim 1, wherein the computer comprises: e) a spectralprocessor for decomposing each image into principal components, f) aspatial processor which receives input from the spectral processor, thespatial processor comprising means for detecting predetermined shapes inthe images, and g) a temporal processor which receives input from thespatial processor, the temporal processor comprising means for analyzingsequences of images and for detecting the presence of predeterminedmovements of said predetermined shapes in said sequences of images. 6.The system of claim 5, wherein said predetermined shapes comprise humanforms, and wherein said predetermined movements comprise verticalmovements and vertical movements with rotation.
 7. A method formonitoring swimming pools for possible drowning victims, the methodcomprising: a) forming a sequence of digital images of a pool, eachimage being formed by a passive sensor positioned above a surface of thepool, b) analyzing said images to determine whether the images indicatea presence of a drowning victim in the pool, and c) alerting an operatorof the presence of a drowning victim, wherein step (b) includescompensating for refraction of images caused by a roughened watersurface due to surface waves in the pool, wherein the water surfacedefines a plurality of areas having different refraction properties. 8.The method of claim 7, wherein step (b) includes analyzing said imagesto locate human forms in the images, and to detect whether said humanforms are displaying movements consistent with possible drowning.
 9. Themethod of claim 7, wherein there are at least two sensors, and whereinstep (b) includes comparing images received by the sensors so as tocompute a distance to an object in said images.
 10. The method of claim7, wherein step (b) includes enhancing quality of images by extractingprincipal components of the images, wherein said principal componentsare non-correlated, and analyzing at least some of said principalcomponents to derive information about the images.
 11. The method ofclaim 7, wherein step (b) includes: d) decomposing each image intoprincipal components, e) analyzing each image component produced in step(d) to detect predetermined shapes in said image component, and f)analyzing sequences of image components obtained from step (e) to detectpredetermined movements of said predetermined shapes in said sequencesof image components.
 12. The method of claim 11, wherein saidpredetermined shapes are selected to be human forms, and wherein saidpredetermined movements are selected from the group consisting ofvertical movements and vertical movements with rotation.
 13. A method ofmonitoring a pool so as to detect possible drowning victims therein, themethod comprising: a) forming a sequence of digital images of a pool,each image being formed by a passive sensor positioned above a surfaceof the pool, b) decomposing each image into principal components so asto enhance quality of said images, c) analyzing at least some principalcomponents obtained from step (b) to detect a presence of predeterminedshapes in said image, d) analyzing sequences of images having been foundto be likely to contain said predetermined shapes, to detectpredetermined movements associated with drowning, wherein the analyzingsteps include compensating for refraction of images caused by aroughened water surface due to surface waves in the pool, wherein thewater surface defines a plurality of areas having different refractionproperties, and e) alerting an operator of the existence of a possibledrowning event when said predetermined movements are detected.